Lessons learned: Development of COVID-19 clinical staging models at a large urban research institution.

Background/Objective: The University of Illinois at Chicago (UIC), along with many academic institutions worldwide, made significant efforts to address the many challenges presented during the COVID-19 pandemic by developing clinical staging and predictive models. Data from patients with a clinical encounter at UIC from July 1, 2019 to March 30, 2022 were abstracted from the electronic health record and stored in the UIC Center for Clinical and Translational Science Clinical Research Data Warehouse, prior to data analysis. While we saw some success, there were many failures along the way. For this paper, we wanted to discuss some of these obstacles and many of the lessons learned from the journey. Methods: Principle investigators, research staff, and other project team members were invited to complete an anonymous Qualtrics survey to reflect on the project. The survey included open-ended questions centering on participants' opinions about the project, including whether project goals were met, project successes, project failures, and areas that could have been improved. We then identified themes among the results. Results: Nine project team members (out of 30 members contacted) completed the survey. The responders were anonymous. The survey responses were grouped into four key themes: Collaboration, Infrastructure, Data Acquisition/Validation, and Model Building. Conclusion: Through our COVID-19 research efforts, the team learned about our strengths and deficiencies. We continue to work to improve our research and data translation capabilities.

Validation of a deep learning, value-based care model to predict mortality and comorbidities from chest radiographs in COVID-19.

We validate a deep learning model predicting comorbidities from frontal chest radiographs (CXRs) in patients with coronavirus disease 2019 (COVID-19) and compare the model's performance with hierarchical condition category (HCC) and mortality outcomes in COVID-19. The model was trained and tested on 14,121 ambulatory frontal CXRs from 2010 to 2019 at a single institution, modeling select comorbidities using the value-based Medicare Advantage HCC Risk Adjustment Model. Sex, age, HCC codes, and risk adjustment factor (RAF) score were used. The model was validated on frontal CXRs from 413 ambulatory patients with COVID-19 (internal cohort) and on initial frontal CXRs from 487 COVID-19 hospitalized patients (external cohort). The discriminatory ability of the model was assessed using receiver operating characteristic (ROC) curves compared to the HCC data from electronic health records, and predicted age and RAF score were compared using correlation coefficient and absolute mean error. The model predictions were used as covariables in logistic regression models to evaluate the prediction of mortality in the external cohort. Predicted comorbidities from frontal CXRs, including diabetes with chronic complications, obesity, congestive heart failure, arrhythmias, vascular disease, and chronic obstructive pulmonary disease, had a total area under ROC curve (AUC) of 0.85 (95% CI: 0.85-0.86). The ROC AUC of predicted mortality for the model was 0.84 (95% CI,0.79-0.88) for the combined cohorts. This model using only frontal CXRs predicted select comorbidities and RAF score in both internal ambulatory and external hospitalized COVID-19 cohorts and was discriminatory of mortality, supporting its potential use in clinical decision making.

Detecting Racial/Ethnic Health Disparities Using Deep Learning From Frontal Chest Radiography.

PURPOSE: The aim of this study was to assess racial/ethnic and socioeconomic disparities in the difference between atherosclerotic vascular disease prevalence measured by a multitask convolutional neural network (CNN) deep learning model using frontal chest radiographs (CXRs) and the prevalence reflected by administrative hierarchical condition category codes in two cohorts of patients with coronavirus disease 2019 (COVID-19). METHODS: A CNN model, previously published, was trained to predict atherosclerotic disease from ambulatory frontal CXRs. The model was then validated on two cohorts of patients with COVID-19: 814 ambulatory patients from a suburban location (presenting from March 14, 2020, to October 24, 2020, the internal ambulatory cohort) and 485 hospitalized patients from an inner-city location (hospitalized from March 14, 2020, to August 12, 2020, the external hospitalized cohort). The CNN model predictions were validated against electronic health record administrative codes in both cohorts and assessed using the area under the receiver operating characteristic curve (AUC). The CXRs from the ambulatory cohort were also reviewed by two board-certified radiologists and compared with the CNN-predicted values for the same cohort to produce a receiver operating characteristic curve and the AUC. The atherosclerosis diagnosis discrepancy, Δvasc, referring to the difference between the predicted value and presence or absence of the vascular disease HCC categorical code, was calculated. Linear regression was performed to determine the association of Δvasc with the covariates of age, sex, race/ethnicity, language preference, and social deprivation index. Logistic regression was used to look for an association between the presence of any hierarchical condition category codes with Δvasc and other covariates. RESULTS: The CNN prediction for vascular disease from frontal CXRs in the ambulatory cohort had an AUC of 0.85 (95% confidence interval, 0.82-0.89) and in the hospitalized cohort had an AUC of 0.69 (95% confidence interval, 0.64-0.75) against the electronic health record data. In the ambulatory cohort, the consensus radiologists' reading had an AUC of 0.89 (95% confidence interval, 0.86-0.92) relative to the CNN. Multivariate linear regression of Δvasc in the ambulatory cohort demonstrated significant negative associations with non-English-language preference (ß = -0.083, P < .05) and Black or Hispanic race/ethnicity (ß = -0.048, P < .05) and positive associations with age (ß = 0.005, P < .001) and sex (ß = 0.044, P < .05). For the hospitalized cohort, age was also significant (ß = 0.003, P < .01), as was social deprivation index (ß = 0.002, P < .05). The Δvasc variable (odds ratio [OR], 0.34), Black or Hispanic race/ethnicity (OR, 1.58), non-English-language preference (OR, 1.74), and site (OR, 0.22) were independent predictors of having one or more hierarchical condition category codes (P < .01 for all) in the combined patient cohort. CONCLUSIONS: A CNN model was predictive of aortic atherosclerosis in two cohorts (one ambulatory and one hospitalized) with COVID-19. The discrepancy between the CNN model and the administrative code, Δvasc, was associated with language preference in the ambulatory cohort; in the hospitalized cohort, this discrepancy was associated with social deprivation index. The absence of administrative code(s) was associated with Δvasc in the combined cohorts, suggesting that Δvasc is an independent predictor of health disparities. This may suggest that biomarkers extracted from routine imaging studies and compared with electronic health record data could play a role in enhancing value-based health care for traditionally underserved or disadvantaged patients for whom barriers to care exist.

Telemedicine Readiness Across Medical Conditions in a US National Representative Sample of Older Adults.

Telemedicine has provided older adults the ability to seek care remotely during the coronavirus disease (COVID-19) pandemic. However, it is unclear how diverse medical conditions play a role in telemedicine uptake. A total of 3379 participants (≥65 years) were interviewed in 2018 as part of the National Health and Aging Trends Study. We assessed telemedicine readiness across multiple medical conditions. Most chronic medical conditions and mood symptoms were significantly associated with telemedicine unreadiness, for physical or technical reasons or both, while cancer, hypertension, and arthritis were significantly associated with telemedicine readiness. Our findings suggest that multiple medical conditions play a substantial role in telemedicine uptake among older adults in the US. Therefore, comorbidities should be taken into consideration when promoting and adopting telemedicine technologies among older adults.

Predicting clinical outcomes among hospitalized COVID-19 patients using both local and published models.

BACKGROUND: Many models are published which predict outcomes in hospitalized COVID-19 patients. The generalizability of many is unknown. We evaluated the performance of selected models from the literature and our own models to predict outcomes in patients at our institution. METHODS: We searched the literature for models predicting outcomes in inpatients with COVID-19. We produced models of mortality or criticality (mortality or ICU admission) in a development cohort. We tested external models which provided sufficient information and our models using a test cohort of our most recent patients. The performance of models was compared using the area under the receiver operator curve (AUC). RESULTS: Our literature review yielded 41 papers. Of those, 8 were found to have sufficient documentation and concordance with features available in our cohort to implement in our test cohort. All models were from Chinese patients. One model predicted criticality and seven mortality. Tested against the test cohort, internal models had an AUC of 0.84 (0.74-0.94) for mortality and 0.83 (0.76-0.90) for criticality. The best external model had an AUC of 0.89 (0.82-0.96) using three variables, another an AUC of 0.84 (0.78-0.91) using ten variables. AUC's ranged from 0.68 to 0.89. On average, models tested were unable to produce predictions in 27% of patients due to missing lab data. CONCLUSION: Despite differences in pandemic timeline, race, and socio-cultural healthcare context some models derived in China performed well. For healthcare organizations considering implementation of an external model, concordance between the features used in the model and features available in their own patients may be important. Analysis of both local and external models should be done to help decide on what prediction method is used to provide clinical decision support to clinicians treating COVID-19 patients as well as what lab tests should be included in order sets.

Predicting Prolonged Hospitalization and Supplemental Oxygenation in Patients with COVID-19 Infection from Ambulatory Chest Radiographs using Deep Learning.

RATIONALE AND OBJECTIVES: The clinical prognosis of outpatients with coronavirus disease 2019 (COVID-19) remains difficult to predict, with outcomes including asymptomatic, hospitalization, intubation, and death. Here we determined the prognostic value of an outpatient chest radiograph, together with an ensemble of deep learning algorithms predicting comorbidities and airspace disease to identify patients at a higher risk of hospitalization from COVID-19 infection. MATERIALS AND METHODS: This retrospective study included outpatients with COVID-19 confirmed by reverse transcription-polymerase chain reaction testing who received an ambulatory chest radiography between March 17, 2020 and October 24, 2020. In this study, full admission was defined as hospitalization within 14 days of the COVID-19 test for > 2 days with supplemental oxygen. Univariate analysis and machine learning algorithms were used to evaluate the relationship between the deep learning model predictions and hospitalization for > 2 days. RESULTS: The study included 413 patients, 222 men (54%), with a median age of 51 years (interquartile range, 39-62 years). Fifty-one patients (12.3%) required full admission. A boosted decision tree model produced the best prediction. Variables included patient age, frontal chest radiograph predictions of morbid obesity, congestive heart failure and cardiac arrhythmias, and radiographic opacity, with an internally validated area under the curve (AUC) of 0.837 (95% CI: 0.791-0.883) on a test cohort. CONCLUSION: Deep learning analysis of single frontal chest radiographs was used to generate combined comorbidity and pneumonia scores that predict the need for supplemental oxygen and hospitalization for > 2 days in patients with COVID-19 infection with an AUC of 0.837 (95% confidence interval: 0.791-0.883). Comorbidity scoring may prove useful in other clinical scenarios.